{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement Linerar Regression Using Matrix Operations\n",
    "\n",
    "#### Objective:Understand how linear regerssion workes by implementing it from scratch suing matrix operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression Cost Function\n",
    "\n",
    "The cost function: ($J(\\theta)$) for linear regression is defined as:[$J(\\theta) = \\frac{1}{2m} \\sum_{i=1}^{m} (h_\\theta(x^{(i)}) - y^{(i)})^2 $] where:\n",
    "\n",
    "($m$) is the number of training examples\n",
    "\n",
    "($h_\\theta(x)$) is the hypothesis (predicted value),which is ($X_b\\cdot\\theta$)\n",
    "\n",
    "(y) is the actual value.\n",
    "\n",
    "\n",
    "\n",
    "### Gradient of the Cost Function\n",
    "\n",
    "To minimize the cost function,we use gradient descent,which requires the gradient(partial derivative) of the cost function with respect to ($\\theta$).The gardient is given by[$\\frac{\\partial J(\\theta)}{\\partial \\theta} = \\frac{1}{m} X_b^T \\cdot (X_b \\cdot \\theta - y)$]\n",
    "\n",
    "Explanation of the Expression:\n",
    "\n",
    "($X_b\\cdot\\theta$):This computes the predicted values for all training examples.\n",
    "\n",
    "($X_b\\cdot\\theta - y$):This computers the error (difference) between the predicted values and actual values.\n",
    "\n",
    "\n",
    "($X_b^T\\cdot(X_b\\cdot\\theta - y)$):This computes the dot product of the transpose of (X_b) and the error vector,which gives the sum of the gradients for all training examples.\n",
    "\n",
    "($\\frac{1}{m}$):This scales the sum of the gradients by the number of training examples to get the average gradient.\n",
    "\n",
    "\n",
    "### Simplified Expression\n",
    "\n",
    "In the provided code,the expression is:[$\\text{gradients} = \\frac{2}{100} X_b^T \\cdot (X_b \\cdot \\theta - y)$]\n",
    "\n",
    "Here,($\\frac{2}{100}$) is used instead of ($\\frac{1}{m}$) because:\n",
    "\n",
    "(m=100) (number of training examples)\n",
    "\n",
    "The factor of 2 comes from the derivative of the sequared error term in the cost function.\n",
    "\n",
    "\n",
    "### Final Expression:\n",
    "\n",
    "Thus,the expression  ($2/100 * X_b.T.dot(X_b.dot(theta) - y)$) correctly computes the gradient of the cost function for linear regerssion,which is used the update the model parameters ($\\theta$) during gradient descent. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters are: (theta) [[4.22215108]\n",
      " [2.96846751]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "# Generate sysnthetic data\n",
    "np.random.seed(0)\n",
    "X=2*np.random.rand(100,1)\n",
    "y=4+3*X+np.random.randn(100,1)\n",
    "\n",
    "#Add bias term(intercept)\n",
    "X_b=np.c_[np.ones((100,1)),X]\n",
    "\n",
    "#Initialize parameters\n",
    "theta=np.random.randn(2,1)\n",
    "\n",
    "#Learning rate and number of iterations\n",
    "learning_rate=0.1\n",
    "n_iterations=1000\n",
    "\n",
    "\n",
    "#Gradient Descent\n",
    "for iteration in range(n_iterations):\n",
    "    gradients=2/100*X_b.T.dot(X_b.dot(theta)-y)\n",
    "    theta=theta-learning_rate*gradients\n",
    "\n",
    "print(\"Parameters are: (theta)\",theta)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-learning",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
